Apparatus and method for automatic ultrasound segmentation for visualization and measurement

ABSTRACT

A system and method for performing ultrasound scans is provided. One embodiment of the ultrasonagraphic system acquires sonogram information from a series of ultrasonic scans of a human subject. The series of ultrasound scans are taken over a portion of interest on the human subject which has their underlying bone structure or other ultrasound discernable organ that is under examination. The data from the series of scans are synthesized into a single data file that corresponds to a three-dimensional (3D) image and/or 3D model of the underlying bone structure or organ of the examined human subject.

PRIORITY CLAIM

This application is a Continuation of, and claims priority to, copendingU.S. application Ser. No. 17/569,797, filed on Jan. 6, 2022, entitledSystems and Methods For Automatic Ultrasound Segmentation ForVisualization And Measurement, which is a Continuation In Part of, andclaims priority to, copending U.S. application Ser. No. 16/813,469,filed on Mar. 9, 2020, entitled Systems and Methods For AutomaticUltrasound Segmentation For Visualization And Measurement, which arehereby incorporated by reference in their entirety for all purposes.

BACKGROUND OF THE INVENTION

In the arts of human body visualization, and in particular visualizationof a human spine, X-ray images and computed tomography (CT) scans havebeen fairly effective in acquiring image data of human body parts, andin particular, human bone structures such as the spine. Magneticresonance imaging (MRI) is another tool to obtain image data of humanbody parts.

However, X-ray machines, MRI machines and CT scanning machines are veryexpensive to acquire and to operate. X-ray images present graphicalinformation on a limited two-dimensional plane. MRI is unsatisfactorilyslow and provides low resolution images of bone structures.

Further, X-ray imaging and CT scanning use ionizing radiations (X-rays)that may be harmful to the human subject, particularly if the humansubject must undergo repeated testing over a long duration of time. Forexample, a human subject suffering from advancing scoliosis (a curvatureof the spine) must, from time to time, be examined to ascertain theextent and/or change in the scoliosis of their spine. Repeated exposureto radiation during periodic examinations may be harmful to such humansubjects.

Other less potentially harmful devices are available for acquiring humansubject information are available. For example, ultrasound devicesproject sound waves into the human subject and detect returning soundwave echoes to generate an image, referred to as a sonogram. Ultrasounddevices used in ultrasonographic systems produce sound waves at afrequency above the audible range of human hearing, which isapproximately 20 kHz. Sound waves between 2 and 18 Mhz are often usedfor ultrasound medical diagnostic applications. At present, there are noknown long-term side effects from interrogating the human body withultrasound waves.

However, an ultrasound scan can cover only a relatively small part ofthe human subject's body with each scan. Further, the sonogram is arelatively narrow image, covering a relatively small cross section ofonly a few inches. And, objects identified in the sonogram may often beblurry. For example, five hundred to one thousand sonogram images mustbe captured to acquire a sufficient amount of image data for analysis ofa full human spine. Accordingly, legacy ultrasound scanners areinadequate for acquiring image information for the human subject's bodywhen a large area of the human subject must be examined, such as thehuman subject's spine, because the sonogram images are too small and alarge number of sonogram images cannot be easily analyzed to arrive atany meaningful information about the condition of the examined humansubject.

Sonogram scan images can be evaluated as a two dimensional (2D) grid ofimage pixels corresponding to the width and the length of a particularsonogram scan image. Each image pixel is defined by an intensity value(i.e., brightness). The intensity value may be associated with aparticular structure that is detected in a sonogram scan. For example,an image pixel with a high value (wherein the image pixel appears as avisually dark pixel) may be associated with bone tissue of the scannedpatient. Image pixels with low intensity values may be associate withother types of soft tissue. Accordingly, sonogram scan images are knownto provide visual indications of a patient's tissue, such as their bonestructure.

The depth of each pixel in a sonogram scan image may be known or may bedeterminable. Accordingly, each pixel in a sonogram scan image and itsassociated intensity can be associated with a voxel in a 3D volume.Since high intensity values of the sonogram scan image pixels may beassociated with bone structure, the voxels in a 3D volume having highintensity values may be used to define a 3D model of the surface of thepatient's bone along the scanned line of the sonogram scan image.

In 3D computer graphics, 3D modeling is referred to as the process ofdeveloping a mathematical representation of any surface of an object(either inanimate or living) in three dimensions via specializedsoftware. The product is called a 3D model. Three-dimensional (3D)models represent a physical body using a collection of points in 3Dspace, connected by various geometric entities such as triangles, lines,curved surfaces, etc. Being a collection of data (points and otherinformation), 3D models can be created manually, algorithmically(procedural modeling), or by scanning. Their surfaces may be furtherdefined with texture mapping. (See Wikipedia, Feb. 25, 2020.)

An example purpose of a scanner is to create a 3D model. This 3D modelconsists of a point cloud of geometric samples on the surface of thesubject. These points can then be used to extrapolate the shape of thesubject (a process called reconstruction). For most situations, a singlescan will not produce a complete model of the subject. Multiple scans,even hundreds, from many different directions are usually required toobtain information about all sides of the subject. These scans have tobe brought into a common reference system, a process that is usuallycalled alignment or registration, and then merged to create a complete3D model. This whole process, going from the single range map to thewhole model, is usually known as the 3D scanning pipeline. (See forexample Wikipedia, Feb. 6, 2020, and Kim et. al; “SLAM-driven RoboticMapping And Registration Of 3D Point Clouds,” 2018; and Lorensen et. al,“The Visualization Toolkit An Object-Oriented Approach To 3D Graphics,”Ed. 4.1, Chapter 5, July 2018.)

Accordingly, there is a need in the arts to more effectively acquiresonogram image data from a human subject using ultrasound devices forgeneration of 3D models of a patient's tissue of interest.

SUMMARY OF THE INVENTION

Embodiments of the ultrasonagraphic system provide a system and methodfor performing ultrasound scans. One embodiment of the ultrasonagraphicsystem acquires sonogram information from a series of ultrasonic scansof a human subject. The series of ultrasound scans are taken over aportion of interest on the human subject which has their underlying bonestructure or other ultrasound discernable organ that is underexamination. The data from the series of scans are synthesized into asingle data file that corresponds to a three-dimensional (3D) imageand/or 3D model of the underlying bone structure or organ of theexamined human subject.

BRIEF DESCRIPTION OF THE DRAWINGS

The components in the drawings are not necessarily to scale relative toeach other. Like reference numerals designate corresponding partsthroughout the several views.

FIG. 1 is a schematic view of an ultrasonagraphic system for examiningand treating spinal conditions.

FIG. 2 is a graphical image of the spine of a test human subject thathas been examined using an embodiment of the ultrasonagraphic system.

FIG. 3 is a graphical image of the spine of a test human subject afterthe AI algorithm of the image processing algorithm module has analyzedparticular spine bones and has added a graphical artifact overlaid overthe image of the spine bone.

FIG. 4 is a conceptual diagram of the human subject showing their spineand their pelvic bone.

FIG. 5 is a conceptual diagram of a 3D or 2D image generated from thecomposite sonogram graphical information presenting an image of thespine and pelvic bone of the human subject.

FIG. 6 is a block diagram of a programmable computing device suitablefor use as part of the image processing system embodied with anultrasonagraphic system.

FIG. 7 is a flow chart depicting a process used by an example embodimentof the ultrasonagraphic system.

FIG. 8 is a schematic view of an alternative embodiment of theultrasonagraphic system for examining a human subject where the opticaltarget is not located on the human subject.

FIG. 9 is a schematic view of an alternative embodiment of theultrasonagraphic system for examining a human subject where the opticaltarget is not used.

FIG. 10 illustrates two image frames of sonogram image information.

FIG. 11 illustrates an alternative embodiment of the ultrasonagraphicsystem that employs an electromagnetic sensor.

FIG. 12 illustrates an alternative embodiment that employs an internalmoveable ultrasonic transmitter within the ultrasound transducer probe.

FIG. 13 a illustrates that the moveable ultrasonic transmitter moves inan arc pattern by some predefined angle between ultrasound scans.

FIG. 13 b illustrates that the moveable ultrasonic transmitter rotates180 degrees by some predefined angle between ultrasound scans.

FIG. 13 c illustrates that the moveable ultrasonic transmitter moveslaterally by some lateral distance between ultrasound scans.

FIG. 13 d illustrates that the moveable ultrasonic transmitter moves bysome predefined angular displacement between ultrasound scans.

DETAILED DESCRIPTION

Embodiments of the ultrasonagraphic system 100 provides a system andmethod for acquiring sonogram information from a series of ultrasonicscans of a human subject. The series of ultrasound scans are taken overa portion of interest on the human subject which has their underlyingbone structure or other ultrasound discernable organ that is underexamination. The data from the series of scans are synthesized into asingle data file that corresponds to a three-dimensional (3D) imageand/or 3D model of the underlying bone structure or organ of theexamined human subject.

The disclosed ultrasonagraphic system 100 will become better understoodthrough review of the following detailed description in conjunction withthe figures. The detailed description and figures provide merelyexamples of the various inventions described herein. Those skilled inthe art will understand that the disclosed examples may be varied,modified, and altered without departing from the scope of the inventionsdescribed herein. Many variations are contemplated for differentapplications and design considerations; however, for the sake ofbrevity, each and every contemplated variation is not individuallydescribed in the following detailed description.

Throughout the following detailed description, examples of variousultrasonagraphic systems 100 are provided. Related features in theexamples may be identical, similar, or dissimilar in different examples.For the sake of brevity, related features will not be redundantlyexplained in each example. Instead, the use of related feature nameswill cue the reader that the feature with a related feature name may besimilar to the related feature in an example explained previously.Features specific to a given example will be described in thatparticular example. The reader should understand that a given featureneed not be the same or similar to the specific portrayal of a relatedfeature in any given figure or example.

The following definitions apply herein, unless otherwise indicated.

“Substantially” means to be more-or-less conforming to the particulardimension, range, shape, concept, or other aspect modified by the term,such that a feature or component need not conform exactly. For example,a “substantially cylindrical” object means that the object resembles acylinder, but may have one or more deviations from a true cylinder.

“Comprising,” “including,” and “having” (and conjugations thereof) areused interchangeably to mean including but not necessarily limited to,and are open-ended terms not intended to exclude additional, elements ormethod steps not expressly recited.

Terms such as “first”, “second”, and “third” are used to distinguish oridentify various members of a group, or the like, and are not intendedto denote a serial, chronological, or numerical limitation.

“Coupled” means connected, either permanently or releasably, whetherdirectly or indirectly through intervening components.

FIG. 1 is a schematic view of an ultrasonagraphic system 100 foracquiring 3D image information and 3D model data for bone structures orother internal organs of a human subject 102. Various examples aredescribed herein in the context of examining and treating spinalconditions by acquiring 3D image information and 3D model data for thespine of the human subject 102. Alternatively, or additionally,two-dimensional (2D) image information and/or 2D model data may begenerated by embodiments of the ultrasonagraphic system 100.

In the non-limiting example application, the ultrasonagraphic system 100of FIG. 1 is configured to enable a practitioner to acquire ultrasoundimages (sonograms) of a patient's spine in real-time with ultrasoundtransducer probe 104. The ultrasonagraphic processor system 106, afterreceiving sonogram image information from a series of ultrasound scans,generates the 3D image information and/or 3D model data of the spinehuman subject 102 without subjecting the patient to potentially harmfulionizing radiation. Further, ultrasonagraphic system 100 of FIG. 1enables a practitioner to acquire images of the outer cortex of apatient's spine with high resolution on a real-time or substantiallyreal-time basis. One skilled in the art appreciates that theultrasonagraphic processor system 106 may be used to generate 3D imageand/or 3D model data for other portions of the human subject 102. Insome examples, the system is optionally configured to stereoscopicallydisplay the images in three dimensions, such as with the 3Dvisualization module and a 3D/2D stereoscopic display 108 shown in FIG.1 .

An example embodiment of the ultrasonagraphic processor system 106comprises an ultrasound interface 110, an ultrasound image dataprocessor 112, at least one image capture device 114, an optical trackerunit 116, an image registration module 118, an image processingalgorithm module 120, a 3D/2D visualization module 122, and a database124. Some embodiments include an optional clock 126. The ultrasoundinterface 110 communicatively couples the ultrasound transducer probe104 to the ultrasonagraphic processor system 106 via a wire-based orwireless connection 128.

In alternative embodiments, the image registration module 118, the imageprocessing algorithm module 120, and/or the 3D visualization module maybe integrated together, and/or may be integrated with other logic. Inother embodiments, some or all of these memory and other datamanipulation functions may be provided by using a remote server or otherelectronic devices suitably connected via the Internet or otherwise to aclient device (not shown). The database 124 may be implemented using anysuitable local and/or remote memory device or system. Depending upon theembodiment, the database 124 may be a dedicated memory system, may bepart of another component or system, and/or may be a distributed localand/or remote memory system. The database 124 may also include otherlogic, modules and/or databases not illustrated or described herein.Other ultrasonagraphic systems 100 may include some, or may omit some,of the above-described components. Further, additional components notdescribed herein may be included in alternative embodiments.

As conceptually illustrated in FIG. 1 , the ultrasonagraphic system 100utilizes optical tracking technology to precisely detect the location ofthe scanned portion of the patient relative to the ultrasound transducerprobe's 104 position in 3D space. The image capture device 114 acquiresimage information in a space around the human subject 102 and theultrasound transducer probe 104. The image information (interchangeablyreferred to herein as a camera image) is acquired in a periodic serialfashion, preferably at a rate of tens or hundreds of image frames persecond. The image capture device 114 may be any suitable device thatperiodically captures still images or that captures video images (knownin the arts to be a time sequenced series of still images). The capturedimage data is then communicated from the image capture device 114 to theoptical tracking unit 116.

Each acquired camera image has an associated time stamp that specifies atime of camera image capture or camera image time of acquisition. Thecamera image capture time may be expressed in real time or by using areference time. Time stamp information may be provided by an internalclock residing in the image capture device(s) 114. Alternatively, theclock 126 may add in time stamp information to the acquired cameraimages as they are being communicated from the image capture device 114to the optical tracker unit 116.

In some embodiments, a plurality of image capture devices 114 may beused to capture camera images in a synchronized fashion. That is, themultiple image capture devices 114 provide concurrently captured cameraimages with the same time stamp.

The optical tracking unit 116, for each acquired image, identifies theone or more optical targets 130 that have been placed on the surface ofthe body of the human subject 102. Also, the optical tracking unit 116,in each acquired image, identifies the one or more optical targets 132that have been placed on the surface of the ultrasound transducer probe104.

Optical targets 130, 132 may be conventional, specially developed, orlater developed optical targets that are discernable by the opticaltracking unit 116. In some examples, the optical targets 130, 132 extendin three dimensions about three coordinate axes and include distinctoptical target portions representing each axis. In other examples, theoptical target 130, 132 extends in three dimensions about six axes andincludes distinct optical targets representing each of the six axes. Theoptical targets 130, 132 may be active, such as by emitting infraredsignals to the optical target, or passive, such as includingretro-reflective markers affixed to some interaction device.

The optical tracking unit 116 then computes or determines the positionof the ultrasound transducer probe 104 relative to the optical target130 in 3D space for the indexed time. The position determination isbased upon the identified relative location of the optical targets 130,132 in the acquired camera image. One skilled in the art appreciatesthat relative location between optical targets 130, 132 can be basedupon their identified location in an image. Further, orientation of theoptical targets 130, 132 can be determined from an analysis of the imageof the optical targets 130, 132. Accordingly, the position andorientation of the ultrasound transducer probe 104 relative to theoptical target 130 can be determined.

Then, the optical tracking unit 116 determines the correspondinglocation on the body of the human subject 102. This determined locationon the human subject 102 is interchangeably referred herein as the timeindexed location information. The time indexed location informationidentifies the location and the time that the ultrasound transducerprobe 104 was on the human subject 102. Any suitable position trackingsystem now known or later developed may be used by the variousembodiments of the ultrasonagraphic system 100 to determine the timeindexed location information.

It is worth noting that that ultrasonagraphic system 100 of FIG. 1 isconfigured to detect the position of the human subject 102 directly bythe optical target(s) 130 positioned on the human subject 102 as opposedto merely detecting the position of a fixed object near the humansubject 102, such as a chest board or other stationary referenceobjects. Accordingly, if during examination the human subject 102 movesor adjusts their position, the time indexed location informationdetermined from later acquired camera images can be correlated with timeindexed location information determined from earlier acquired cameraimages.

Additionally, or alternatively, to the optical tracking technologyincluded in the example of FIG. 1 , the ultrasonagraphic system 100 mayinclude magnetic positioning systems or attitude heading referencesystems to detect the position of the human subject 102, the ultrasoundtransducer probe 104, or both. For example, location and/or orientationof the ultrasound transducer probe 104 may be determined by variousmicro electro-mechanical devices (MEMS) such as accelerometers or thelike.

Additionally, or alternatively, the ultrasonagraphic system 100 mayinclude an infrared scanning system configured to scan illuminatedobjects, such as the human subject 102, in three-dimensions. Theinfrared scanning system may include an infrared light projector, acamera or CMOS image sensor to detect the infrared light interactingwith illuminated objects, and a microchip including computer executableinstructions for spatially processing scanned objects. Suitable infraredscanning systems include the Light Coding™ system included in theKinect™ gaming system. The infrared scanning system may supplement theoptical tracking device and optical targets described above or mayreplace them in some applications.

In practice, an operator (not shown) such as an ultrasound technician, adoctor, or another individual, operates the ultrasound transducer probe104 in a manner that emits sonic waves 134 into the human subject 102.Within the context of acquiring echo return data, interchangeablyreferred to herein as sonogram information or sonogram data, theoperator begins the scanning process by performing a first sonogram scan136 over a selected location of the human subject 102. To conceptuallyillustrate use of an embodiment of the ultrasonagraphic system 100,examination of the spine of the human subject 102 is described. Thesonogram scanning begins at a first location on the human subject 102,such as near the head of the human subject 102, and that is at alocation that is to the side of the centerline of the spine of the humansubject 102. The operator then moves the ultrasound transducer probe 104in a substantially straight line across the spine of the human subject102.

During the first sonogram scan 136, in an example embodiment, sonograminformation corresponding to a plurality of serially acquired sonogramimages are communicated from the ultrasound transducer probe 104 to theultrasound interface 110. A time stamp corresponding to the time ofacquiring the sonogram image is added by ultrasound transducer probe 104to an individual sonogram image to generate time indexed sonogram imageinformation portion. Alternatively, the clock 126 may add the timeinformation to the acquired sonogram image to generate the time indexedsonogram information portion.

Alternatively, the ultrasound transducer probe 104 may provide acontinuous stream of sonogram information (echo return data)corresponding to the return echoes detected by the ultrasound transducerprobe 104 during each scan. When a stream of data corresponding todetected echoes is provided, time stamps are periodically added intoand/or are associated with particular potions of the streaming echoreturn data. Thus, the echo return data acquired during the beginning ofthe scan will have an associated first time stamp that corresponds tothe time of data acquisition, and later portions of the acquired echoreturn data will be associated with later time stamps to reflect thetime that that echo return data was acquired by the ultrasoundtransducer probe 104.

The associated time stamp specifies a time of acquisition of thesonogram image and/or acquisition of a portion of the stream of sonogramecho data. The time indexed sonogram image and/or the time indexedsonogram echo data portion is interchangeably referred to herein as thetime indexed sonogram image information portion.

The time stamps associated with the time indexed sonogram informationmay be expressed in real time or by using a reference time. The timestamp information may be provided by an internal clock residing in theultrasound transducer probe 104. Alternatively, the clock 126 may add intime stamp information to the acquired sonogram information as the timeindexed sonogram information is communicated from the ultrasoundinterface 110 to the ultrasound image data processor 112. The sonogramimage time stamps have the same time reference as the corresponding timestamps associated with the time indexed camera images concurrentlycaptured by the image capture device 114.

As is known in the arts, the ultrasound image data processor 112processes the received sonogram information acquired during the firstsonogram scan 136 into a time indexed sonogram image informationportions which may be used to render the first sonogram image. The timeindexed sonogram image information portions are communicated from theultrasound image data processor 112 to the image registration module118.

The time stamps of each of a plurality of portions of the first sonogramimage are correlated with the corresponding time indexed camera imagesby the image registration module 118. For each of the time stamps of oneof the time indexed sonogram image information portions, a correspondingcamera image with the same or substantially the same time stamp iscorrelated with that particular time indexed sonogram image informationportion. Accordingly, the location and orientation of the ultrasoundtransducer probe 104 relative to the target 130 on the human subject 102during each portion of the first sonogram scan 136 is determined. Thatis, the location and orientation of the ultrasound transducer probe 104,and therefore the location of each time indexed sonogram imageinformation portion on the body of the human subject 102 is determinedby the image registration module 118. The location on the body of thehuman subject 102 is based on the location information determined fromthe corresponding time indexed camera image that has the same time stampinformation. Accordingly, the location information for each associatedlocation indexed sonogram image information portion identifies thelocation of that first sonogram image portion on the body of the humansubject 102.

As one skilled in the arts appreciates, a first sonogram imagesgenerated from the first sonogram scan 136 has a relatively narrow range(width) that typically only encompasses one or more inches of width.Accordingly, after the first sonogram scan 136 has been completed, theoperator shifts the position of the ultrasound transducer probe 104downward by a predefined incremental distance (referred to herein as the“sonogram scan shift distance”) for a subsequent sonogram scan.Preferably, the sonogram scan shift distance is no greater than thesonogram image width of the sonogram images acquired during the firstsonogram scan 136. The operator then conducts a second sonogram scan 138across the human subject 102.

The second sonogram scan 138 runs parallel to, or substantially parallelto, the first sonogram scan 136. One skilled in the art appreciates thatsome degree of overlap between the sonogram image information acquiredduring the second sonogram scan 138 and the first sonogram imageinformation acquired during the first sonogram scan 136 may occur. Insome situations, such an overlap in the sonogram image information isdesirable. During later construction of the 3D/2D image and/or 3D/2Dmodel data, information determined from any overlapping portions ofsonogram image information is merely duplicative and can be discarded,erased, or is not used, and therefore, will not adversely impactgeneration of the 3D/2D image and/or 3D/2D model data. In someembodiments, the duplicative information is combined to generateenhanced sonogram image information.

The image registration module 118 then generates a plurality of locationindexed sonogram image information portions based on the informationreceived from the ultrasound image data processor 112 and the opticaltracking unit 116. For each processed sonogram scan, the locationindexed sonogram image information portions comprise sonogram imageinformation for the particular portion of the sonogram scan, optionaltime indexing information where each time index identifies a particulartime of acquisition for the associated sonogram image portion, andlocation information for each associated sonogram image portion thatidentifies the location of the image portion on the body of the humansubject 102.

Similarly, a third sonogram scan 142, adjacent to the second sonogramscan 138, may be acquired. The process of conducting a continuing seriesof sonogram scans 140 continues, with each successive sonogram scanbeing separated by the previous sonogram scan by the predefined sonogramscan shift distance. The process of conducting the series of parallelsonogram scans continues over the portion of the human subject 102 thatis being examined. In the illustrative example of examining the spine ofthe human subject 102, the sonogram scanning process may end with thelast sonogram scan 144 that corresponds to a scan of the lower end ofthe spine of the human subject 102.

The scanning process above was described as a parallel series ofsonogram scans 136 to 144 which were oriented perpendicular to theorientation of the spine of the human subject 102. This scanningsequence is convenient because the operator of the ultrasound transducerprobe 104 can intuitively keep track of their sonogram scans that theyhave performed during the examination of the human subject 102. Onskilled in the arts appreciates that any sonogram scanning process maybe used during examination of the human subject 102 because theorientation and location of the ultrasound transducer probe 104 withrespect to the scan location on the human subject 102 is readilydeterminable. For example, sonogram scan can be aligned along a diagonalto the examined area of the human subject 102. Criss-crossing sonogramscans may be used for the examination, Even elliptical or circularscanning motions may be used during an examination. Such varyingsonogram patterns, by the end of the examination process, will al resultin generation of complete 3D image and/or 3D data for the examined areaof the human subject 102.

The image processing algorithm module 120 receives the plurality oflocation indexed sonogram image information portions that are generatedfor each sonogram scan from the image registration module 118. Thereceived location indexed sonogram image information portions for eachsonogram scan are stored in a suitable memory medium (not shown) or inthe database 124. In an example embodiment, each subsequently receivedlocation indexed sonogram image information portion is stored during thescanning process.

At the conclusion of the scanning process wherein the last locationindexed sonogram image information portions generated from the lastsonogram scan 144 is received, the location indexed sonogram imageinformation for each individual sonogram scan is retrieved by the imageprocessing algorithm module 120 for processing. The processingencompasses a plurality of processing steps.

An initial processing step performed by the image processing algorithmmodule 120 is to aggregate or combine the plurality of individuallocation indexed sonogram image information portions into compositesonogram image information. When the sonogram information and/or data isprovided in discrete image files, individual image frames are selectedfor processing. When the sonogram information is provided as acontinuous stream of data, the streaming echo return data is parsedusing a suitable sampler algorithm into sonogram image portions. Forexample, one slice or frame may be taken from the streaming echo returndata every 0.1 seconds and then saved for further processing. Anysuitable sampling time may be used. Any suitable sampling applicationnow known or later developed that transforms a continuous-time signal toa discrete time signal may be used by embodiments of theultrasonagraphic system 100.

Since each of the individual location indexed sonogram image informationportions are referenced to a reference location on the body of the humansubject 102 (the reference location is determined from the location ofthe marker 130), each portion of the location indexed sonogram imageinformation portions can be ordered by its particular location on thebody of the human subject 102, and then may be combined with (orstitched together) with adjacent portions of the previously acquiredlocation indexed sonogram image information portions and thesubsequently acquired location indexed sonogram image informationportions. For example, the second location indexed sonogram imageinformation portion generated from the second sonogram scan 138 iscombined with the adjacent previously generated first location sonogramimage information portion (generated from the first sonogram scan 136).And, the second location indexed sonogram image information portiongenerated from the second sonogram scan 138 is combined with theadjacent previously generated third location sonogram image informationportion (generated from the third sonogram scan 140). This combining oflocation indexed sonogram image information portions continues until allof the generated location indexed sonogram image information portionsare combined into a single composite sonogram image information file ordata. Any suitable methodology of combining together the locationindexed sonogram image information portions, referred to in the arts asimage stitching, that is now known or later developed, may be used byembodiments of the ultrasonagraphic system 100.

An alternative embodiment generates the composite sonogram imageinformation by combining each received location indexed sonogram imageinformation portions into the composite sonogram image information asthey are received from the image registration module 118. Here, thecomposite sonogram image information is being generated in real time, orin near real time. As described herein, graphical presentation of thecomposite sonogram image information may be presented to the operator aseach of the serial individual sonogram scans are performed. The “size”of the displayed graphical presentation of the composite sonogram imageinformation will increase as each successive sonogram scan is beingperformed. Such immediate real time, or near real time, feedback may beparticularly desirable to assist the operator in obtaining completecoverage of the portion of the body of the human subject 102 that isbeing examined. That is, if a portion of the body is missed in a scan,or the image information is unclear or corrupted, the operator mayrescan the portion of the body of the human subject 102 that is ofinterest such that the subsequently acquired composite sonogram imageinformation portions are integrated into the composite sonogram imageinformation.

So long as the location of the marker 130 on the human subject 102 hasnot changed, additional sonogram scans may be performed by the operatorafter the last sonogram scan 144 has been performed. The subsequentlyacquired sonogram image information can then be correlated withpreviously acquired sonogram scan information. Thus, if a missed portionis later identified, and/or if additional image data for a particularregion on the body is desired for clarity and/or improved resolution,the operator may re-scan that particular region of the body of the humansubject 102. The subsequently acquired composite sonogram imageinformation is then integrated into the previously generated compositesonogram image information. Because the ensuing sonogram scan is bothtime and location indexed, the subsequent scan does not need to be inparallel with the original sonogram scans. The subsequent sonographicscan(s) may be made along any direction of interest.

Once the composite sonogram image information has been generated, thecomposite sonogram image information may be stored into the database124. The database 124 may be located locally, or may be remotelylocated. Once stored, the composite sonogram image information may beretrieved at a later time for processing.

Additionally, the image processing algorithm module 120 processes thecomposite sonogram image information into composite sonogram graphicalinformation that can be used to render a 3D image and/or a 2D image. Thecomposite sonogram graphical information may be optionally saved intothe database 124.

Alternatively, or additionally, the generated composite sonogramgraphical information may be communicated to the 3D/2D visualizationmodule 122. The 3D/2D visualization module 122 processes (renders) thereceived composite sonogram graphical information into image informationthat can be communicated to the 3D/2D stereoscopic display 108, or toanother suitable display device, for presentation to the operator orother individual. Any suitable image rendering process now known orlater developed may be used by the 3D/2D visualization module 122 togenerate presentable composite sonogram images. Alternatively, oradditionally, the composite sonogram graphical information may becommunicated to a remote display system 146 that is configured to renderand present the composite sonogram images.

One skilled in the art appreciates that once the 3D composite sonogramgraphical information has been generated from the composite sonogramimage information, any suitable 3D image presentation algorithm may beused to display the body part of interest of the examined human subject102. The graphical image, here the example spine of the human subject102, may be rotated and/or oriented in any manner for view by theoperator or another individual, such a s specialist doctor. Any suitable3D processing algorithm now known or later developed may be used topresent images generated from the composite sonogram graphicalinformation.

Other analysis algorithms may be integrated into the image processingalgorithm module 120, and/or work in conjunction with, the imageprocessing algorithm module 120. For example, in the context ofassessing degrees of scoliosis in the human subject 102, a spinemodelling and measurement algorithm may be used to perform automaticmeasurement and analysis of the spine of the examined human subject 102.

Preferably, but not required, when the ultrasonagraphic system 100 isused to examine bone structure of the human subject 102, the imageprocessing algorithm module 120 includes a filtering algorithm thatfilters non-bone type sonogram echo information out from the compositesonogram image information (or the plurality of location indexedsonogram image information). Here, background information in thedetected sonic echo information is suppressed so that only echoes fromthe bone structure of the human subject 102 is retained for analysis.

Alternatively, or additionally, other filtering algorithms may be usedto identify and isolate other tissues or structures of interest in theexamined human subject 102. For example, if images of a particular softtissue in the human subject 102 is of interest, echo informationgenerated by nearby bones may be suppressed by a filtering algorithmsuch that the composite sonogram image information is filtered toproduce 3D and/or 2D graphical images of the tissue of interest (organsof interest).

Use of ultrasound transducer probes 104 has not provided satisfactorybecause the sonogram images were too noisy so that a high degree ofdiscrimination of particular organs and/or anatomy of interest simplywas not possible. The novel approach of applying a particular filter tothe sonogram information prior to attempting to construct the 3D imageand/or 3D data now enables embodiments of the ultrasonagraphic system100 to discern particular organs of interest. Here, the exampleapplication described an examination of the spine of the original humansubject 102. The applied filter is configured to filter out sonograminformation that is unrelated to the spine bones of interest.

Preferably, but not required, an artificial intelligence (AI) algorithmmay be used by the image processing algorithm module 120 to enhance thequality and/or reliability of the generated composite sonogram imageinformation. Convolutional neural networks (CNNs) may be used for imageprocessing by embodiments of the ultrasonagraphic system 100. The AIalgorithm learns to further process the received filtered sonograminformation to provide an improved highlight of the organ or anatomy ofinterest that is being examined. For example, but not limited to, the AIsystem may learn to identify particular anatomical landmarks ofinterest. Any suitable AI systems and/or neural networks now known orlater developed may be used by the various embodiments of theultrasonagraphic system 100.

For example, if particular anatomical landmarks on the spine of thehuman subject 102 are of interest, information identifying suchlandmarks may be provided to the image processing algorithm module 120,and in particular to the AI algorithm. Over time, as a greater number oflike landmarks are identified by the AI algorithm in other compositesonogram image information acquired from other human subjects 102, oreven later acquired sonogram information from the original human subject102, the AI algorithm may learn to identify the anatomical landmarks ofinterest.

For example, to assess the position and orientation of the spine, it iscurrently necessary to interact with the 3D reconstruction whengenerating the 3D image and/or 3D model data. This process of manuallyadding points can take time and can be less than perfect. To speed ofthe process, embodiments of the ultrasonagraphic system 100 trains an AImodel that can assess the 3D reconstruction volume in its entirety andassign label to an area by coloring it or other methods. The generated3D image and/or 3D model data may be marked by hand and then the AI istrained to identify the landmark in a similar pattern on a different 3Dvolume. The landmarks that are detected can be identified and analyzedafter the volume is collected or live while the 3D volume is being madein real-time.

In a preferred embodiment, the AI algorithm may optionally be configuredto compute and/or acquire measurement information pertaining to aparticular anatomical structure of interest. In the case of anexamination of the spine of the human subject 102, an example embodimentassess orientation and/or location of particular spine bones. That is, adegree of curvature, rotation and/or tilt between individual spine bones(vertebrae) can be automatically determined. Measurements of the angulardisplacement and/or the location displacement of the spine bones can bedetermined. Embodiments of the ultrasonagraphic system 100 may have theAI algorithm learn to determine any anatomical measurement of interestfor any organ and/or anatomy of interest that is being examined.

Further, when the graphical presentation of the composite sonogram imageinformation is made on the 3D or 2D display 108, the AI algorithm maymodify the graphical image to generate an annotated image that presentsone or more graphical artifacts to highlight and/or present indicatinginformation to the operator or other interested person viewing thepresented graphical image. In the context of the examination of thespine of the human subject 102, the generated graphical artifacts can beused to provide a visual indication of the determined measurements ofthe angular displacement and/or the location displacement of the spinebones.

FIG. 2 is a graphical image 200 of the spine of a test human subject 102that has been examined using an embodiment of the ultrasonagraphicsystem 100. The image 200 presents a graphical image of a portion of thespine of an examined human subject 102. The indicated region 202 informsthe operator or other interested party that the very light areas of theimage portion 202 is showing a sonographic image a particular bone inthe spine of the examined human subject 102. The indicated region 204similarly indicates that the very light areas of the image portion 204is showing a sonographic image of a different bone in the spine of theexamined human subject 102. The indicated region 206 is also a verylight area, but due to a lighter region 208 region 206 is notcharacteristic of bone. Region 206 is likely a ligament or musclesurface connecting bones corresponding to regions 202 and 206. Theindicated regions like 210 similarly indicate that the darker shadedareas of the image portion are showing a sonographic image correspondingto regions of bone surface under the ultrasound transducer probe 104.The bone surfaces reflect most of the ultrasound, causing darker areasunder the spine of the examined human subject 102.

FIG. 3 is a graphical image 300 of the spine of a test human subject 102after the AI algorithm of the image processing algorithm module 120 hasanalyzed particular spine bones and has added one or more graphicalartifacts overlaid over the image of the spine of the human subject 102.The graphical artifacts impart information to the operator or otherinterested party regarding the state or condition of the associatedspine. For example, a single presented image of the spine of the humansubject 102 may be measured to ascertain spinal curvatures in alldirections and planes. Additionally, or alternatively, a selected planeof a 3D image and/or 3D data may be used to generate a 2D image and/or2D data along the plane of interest.

The absence of graphical artifacts to the regions 306, 308 and 310(which correspond to the regions 206, 208 and 210, respectively, of FIG.2 ) indicates that these regions of the examined human subject 102 haveno information of particular interest, and/or were associated with othertissues that were not under examination. Accordingly, no graphicalartifacts were generated for these regions 306, 208 or 310.

FIG. 4 is a conceptual diagram 400 of the human subject 102 showingtheir spine 402 and their pelvic bone 404. FIG. 5 is a conceptualdiagram 500 of a 3D or 2D image generated from the composite sonogramgraphical information presenting an image of the spine 402 and pelvicbone 404 of the human subject 102. The image 500 illustrates a firstgraphical artifact 502 (corresponding to the graphical artifact 302 ofFIG. 3 ) and a second graphical artifact 504 (corresponding to thegraphical artifact 304 of FIG. 3 ). There, the AI algorithm of the imageprocessing algorithm module 120 has presented useful diagnosticinformation to the viewing operator or other interested party.

In some embodiments, graphical artifacts may be generated to highlightparticular anatomical features of interest to aid the assessment of thecondition of the examined human subject 102. For example, a plurality ofgraphical artifacts 506 highlighting the outward protruding portions ofeach spine bone (bumps) can be generated and then overlaid over thegenerated image of the spine. Various colors, shading and/orillumination intensities may be used in the presented graphicalartifacts 506 to aid the examiner in assessing the condition of thespine of the human subject 102. In this simplified conceptual example,the graphical artifacts 506 were generated and presented for only fourspine bones. However, the graphical artifacts 506 may be generated andpresented for all of the spine bones, or for selected spine bones ofinterest. Further, with an interactive display 108 and a suitablegraphical user interface (GUI), the examiner may interactively selectand/or manipulate any presented graphical artifacts 502, 504, and/or506.

The 3D or 2D image of the spine is generated by the image processingalgorithm module 120 during the further processing of the compositesonogram image information. Here, after extensive filtering, imageinformation or data identifying the particular bones in the spine of thehuman subject 102 are identified with a high degree of discrimination.Further, the AI algorithm may have learned to identify particular bonesof the spine of the human subject 102. The AI may access suitable skins(image data that graphically depict a more realistic image of a bone)for each particular bone in the spine of the human subject 102, and usethe accessed skins to create a more realistic graphical representationof the spine.

In some embodiments, a corresponding ideal structure image may beaccessed from a database and overlaid over the top of, or presentedadjacent to, the image generated based on the sonogram examination. Forexample, image data of an ideal spine may be accessed. The image data ofthe ideal spine may then be scaled to correspond to the image of thespine of the human subject 102. Then the overlaid or adjacent image ofthe ideal spine can be visually compared with the image of the spine ofthe examined human subject 102. The comparison may be performed using 3Dor 2D images.

FIG. 6 is a block diagram of a programmable computing device suitablefor use as part of the image processing system 600 embodied with anultrasonagraphic system 100. While the following paragraphs describe onesuitable example of an image processing system, one skilled in the artwill understand that many different examples are contemplated. Forexample, image processing system 600 could include an embedded softwaresystem, a standalone personal computer, and/or a networked computersystem.

From the disclosure of the ultrasonagraphic system 100, those skilled inthe art will recognize that various examples of the image processingsystem 600 may be implemented using electronic circuitry configured toperform one or more functions. For example, with some embodiments of theinvention, the image processing system may be implemented using one ormore application-specific integrated circuits (ASICs). In some examples,however, components of various examples of the invention will beimplemented using a programmable computing device executing firmware orsoftware instructions, or by some combination of purpose-specificelectronic circuitry and firmware or software instructions executing ona programmable computing device.

Accordingly, FIG. 6 shows one illustrative example of an imageprocessing system 600, a computer, that can be used to implement variousembodiments of the invention. As seen in this figure, the example imageprocessing system 600 has a computing unit 602. Computing unit 602typically includes a processing unit 604 and a system memory 606.Processing unit 604 may be any type of processing device for executingsoftware instructions, but will conventionally be a microprocessordevice. System memory 606 may include both a read-only memory (ROM) 608and a random access memory (RAM) 610. As will be appreciated by those ofordinary skill in the art, both read-only memory (ROM) 608 and randomaccess memory (RAM) 610 may store software instructions to be executedby processing unit 604.

Processing unit 604 and system memory 606 are connected, either directlyor indirectly, through a bus 612 or alternate communication structure toone or more peripheral devices. For example, processing unit 604 orsystem memory 606 may be directly or indirectly connected to additionalmemory storage, such as a hard disk drive 614, a removable optical diskdrive 616, a removable magnetic disk drive 618, and a flash memory card620. Processing unit 604 and system memory 606 also may be directly orindirectly connected to one or more input devices 622 and one or moreoutput devices 624. Input devices 622 may include, for example, akeyboard, touch screen, a remote control pad, a pointing device (such asa mouse, touchpad, stylus, trackball, or joystick), a scanner, a cameraor a microphone. Output devices 624 may include, for example, a monitordisplay, an integrated display, television, printer, stereo, orspeakers.

Still further, computing unit 602 will be directly or indirectlyconnected to one or more network interfaces 626 for communicating with anetwork. This type of network interface 626, also sometimes referred toas a network adapter or network interface card (NIC), translates dataand control signals from computing unit 602 into network messagesaccording to one or more communication protocols, such as theTransmission Control Protocol (TCP), the Internet Protocol (IP), and theUser Datagram Protocol (UDP). These protocols are well known in the art,and thus will not be discussed here in more detail. An interface 626 mayemploy any suitable connection agent for connecting to a network,including, for example, a wireless transceiver, a power line adapter, amodem, or an Ethernet connection.

It should be appreciated that, in addition to the input, output andstorage peripheral devices specifically listed above, the computingdevice may be connected to a variety of other peripheral devices,including some that may perform input, output and storage functions, orsome combination thereof. For example, the computer 101 will often beconnected to the 3D ultrasound processor and transducer system. Inaddition to a 3D ultrasound unit, computer 101 may be connected to orotherwise include one or more other peripheral devices, such as atelephone, facsimile machine, router or the like.

The telephone may be, for example, a wireless “smart phone,” such asthose featuring the Android or iOS operating systems. As known in theart, this type of telephone communicates through a wireless networkusing radio frequency transmissions. In addition to simple communicationfunctionality, a “smart phone” may also provide a user with one or moredata management functions, such as sending, receiving and viewingelectronic messages (e.g., electronic mail messages, SMS text messages,etc.), recording or playing back sound files, recording or playing backimage files (e.g., still picture or moving video image files), viewingand editing files with text (e.g., Microsoft Word or Excel files, orAdobe Acrobat files), etc. Because of the data management capability ofthis type of telephone, a user may connect the telephone with computingunit 602 so that their data maintained may be synchronized.

Of course, still other peripheral devices may be included with orotherwise connected to a computing unit 602 of the type illustrated inFIG. 2 , as is well known in the art. In some cases, a peripheral devicemay be permanently or semi-permanently connected to computing unit 602.For example, with many computers, computing unit 602, hard disk drive614, removable optical disk drive 616 and a display are semi-permanentlyencased in a single housing.

Still other peripheral devices may be removably connected to computingunit 602, however. Computing unit 602 may include, for example, one ormore communication ports through which a peripheral device can beconnected to computing unit 602 (either directly or indirectly throughbus 612). These communication ports may thus include a parallel bus portor a serial bus port, such as a serial bus port using the UniversalSerial Bus (USB) standard or the IEEE 1394 High Speed Serial Busstandard (e.g., a Firewire port). Alternately or additionally, computer101 may include a wireless data “port,” such as a Bluetooth® interface,a Wi-Fi interface, an infrared data port, or the like.

It should be appreciated that a computing device employed according tovarious examples of the invention may include more components otherthan, or in addition to, the computing unit 602 illustrated in FIG. 6 .Further, fewer components than computing unit 602, or a differentcombination of components than computing unit 602, may be used byalternative embodiments. Some implementations of the invention, forexample, may employ one or more computing devices that are intended tohave a very specific functionality, such as a server computer. Thesecomputing devices may thus omit unnecessary peripherals, such as thenetwork interface 626, removable optical disk drive 616, printers,scanners, external hard drives, etc. Some implementations of theinvention may alternately or additionally employ computing devices thatare intended to be capable of a wide variety of functions, such as adesktop or laptop personal computer. These computing devices may haveany combination of peripheral devices or additional components asdesired.

Further, in another practice example, the ultrasonagraphic system 100can be used to recall previously generated images and/or data of thespine of the human subject 102 at a later time. As noted herein, theprevious images and/or data can be stored in the computer system of theultrasonagraphic system 100 and/or a local or remote database. Theultrasonagraphic system 100 then matches current 3D images of the spineof the human subject 102 with previously acquired images. In thisexample application, the compared images can be used to assist theoperator in assessment spinal health and/or treatment effectiveness forthe spine of the human subject 102.

Embodiments of the ultrasonagraphic system 100, in addition to obtainingimage information and generated 3D or 2D model information or data forscanned tissues, bones or organs of a human subject 102, may be suitedfor obtaining sonogram information from other animals, such as, but notlimited to pets, livestock, zoo animals, or the like. Embodiments of theultrasonagraphic system 100 may also be used to obtain sonograminformation from plants or other inanimate objects. For example, ancientartifacts or relics that are suitable for scanning using an ultrasoundtransducer probe could be scanned using an embodiment of theultrasonagraphic system 100. Further, sonograph scans are often used toscan prenatal infants while in their mother's womb. Embodiments of theultrasonagraphic system 100 could also be used to scan these infants.

One skilled in the arts appreciates that embodiments of theultrasonagraphic system 100 may also be configured to detect,discriminate, identify and then present other non-biological objectsthat are within the human subject 102. For example, metallic or polymerpins, screws, braces or the like may have been implanted into the humansubject 102 during prior surgical procedures. Such objects can beidentified and then added into the generated 3D or 2D model informationor data. Further, since the 3D or 2D model information or data can begenerated in real time, or in near real time, surgical instrumentscurrently used during a procedure that the human subject 102 isundergoing can be identified. Here, the ultrasonagraphic system 100 canbe used to concurrently detect such surgical instruments along with theorgans or tissue of interest.

One skilled in the art appreciates that the ultrasonagraphic processorsystem 106 does to need to be local to or in proximity to the ultrasoundtransducer probe 104 during examination of the human subject 102. Here,the image capture device 114 needs to be local to the human subject 102during the sonogram scanning process so as to capture images of thetargets 130, 132 during the examination of the human subject 102. Suchembodiments may be configured to be remotely located from the ultrasoundtransducer probe 104 and the image capture device(s) 114. Theultrasonagraphic processor system 106 receives the ultrasoundinformation and the camera images via a suitable communication systemthat communicatively couples the ultrasonagraphic processor system 106with the ultrasound transducer probe 104 and the image capture device(s)114. Further, such embodiments may be configured to receive informationfrom multiple ultrasound transducer probes 104 and image capturedevice(s) 114 so that multiple human subjects 102 may be remotely and/orconcurrently examined. Further, the generated 3D/2D image and/or 3D/2Dmodel data may be communicated back to the examination site for displayon a display device that is located in at the examination site.

FIG. 7 is a flow chart depicting a process used by an example embodimentof the ultrasonagraphic system 100. Camera images and a stream ofsonogram image information are acquired during a sonogram scan (602). Atime stamp is added to the camera image to generate a time indexedcamera image (604). The same, or substantially the same, time stamp isadded to the corresponding portion of the sonogram image information togenerate a time indexed sonogram image information portion.

Each time indexed camera image is processed to determine a correspondingtime indexed location information that identifies the particular portionof the human subject 102 that was being scanned during the time of thetime stamp (606).

Concurrently, a stream of sonogram image information is acquired duringa plurality of sonogram scans (608). A time stamp is added to portionsof the sonogram image information to generate a time indexed sonogramimage information portion (610).

Then, for a selected time (612), the time indexed location informationfor the selected time is combined with the time indexed sonogram imageinformation portion (that was acquired at the same time of the timestamp) to generate a location indexed sonogram image information portionfor that particular portion of the sonogram information (614).

Then, each of the plurality of location indexed sonogram imageinformation portions are combined to generate the composite sonogramimage information (616). The composite sonogram image information isthen used to generate 3D or 2D composite sonogram graphical information(618) which may be used to render 3D or 2D images on a display (620).

Alternative embodiments may employ other methods and apparatus oftracking the position and orientation of the ultrasound transducer probe104 used by the ultrasonagraphic system 100 to identify tissues (organs)and bones of the human subject 102 (FIG. 1 ).

FIG. 8 is a schematic view of an alternative embodiment of theultrasonagraphic system 100 for examining a human subject 102 where theoptical target 130 is not located proximate to, but not on, the humansubject 102. In this non-limiting example embodiment, the optical target130 is placed on a stationary object (rather than the human subject102). Alternatively, a stationary object proximate to the human subject102 that is identifiable in a captured image may be used instead of theoptical target 130.

FIG. 9 is a schematic view of an alternative embodiment of theultrasonagraphic system 100 for examining a human subject 102 where theoptical target 130 is not used. Here, only the position and orientationof the ultrasound transducer probe 104 is tracked during the series ofultrasonic scans.

Using the embodiments illustrated in FIGS. 8 and 9 , the human subject102 remains still, or at least substantially still, during the scanningprocess. The ultrasonagraphic system 100 receives the sonogram imageinformation from the ultrasound transducer probe 104 during a series ofultrasonic scans. The spatial relationship between the scanned tissue ofinterest and the acquired sonogram image information is determinable forgeneration of the 3D image and/or 3D model data.

With these embodiments, the practitioner may view the generated 3D imageand/or 3D model data on the display 108. Based on the experience of thepractitioner, and the approximate known area of the human subject 102that was scanned, the practitioner can make an intuitive judgement toidentify what tissue or bone structure is being shown in the viewed 3Dimage and/or 3D model data.

An alternative embodiment employs a multi-kernel block matchingalgorithm. FIG. 10 illustrates two image frames of sonogram imageinformation. A block matching algorithm divides a first selected imageframe 1002 into macroblocks, wherein a first macroblock 1004 isselected. Preferably, the selected first macroblock 1002 includes areadily identifiable graphical artifact 1006.

A macroblock 1008 corresponding to the first macroblock 1004 isidentified in the second selected image frame 1010. The correspondingmacroblock 1008 is identifiable since it also includes the graphicalartifact 1006.

The two macroblocks 1004, 1008 are compared to define a distance oftravel corresponding to motion of the ultrasound transducer probe 104during the ultrasonic scan. Time information that identifies the timethat the first and second sonogram image information was acquired can becompared to determine a time of capture between the two macroblocks1004, 1008. A vector is then determined that models the movement of amacroblock from one location to another location. This determinedmovement of the macroblocks 1004, 1008 constitutes a motion estimationof the ultrasound transducer probe 104.

As the practitioner conducts a series of ultrasonic scans over the humansubject 102, the graphical artifact 1006 may be identified in the imagesof a subsequent ultrasonic scan. Based on the location of the graphicalartifact 1006 in the sonogram image information of the subsequent scan,the location relationship between the sonogram image information of thepreceding ultrasonic scan and the subsequent ultrasonic scan can becomputed.

As the ultrasonic scanning process continues, at some juncture thegraphical artifact 1006 may not be present in the sonogram imageinformation acquired during subsequent ultrasonic scans. However, adifferent graphical artifact may have been identified in at least onethe subsequent ultrasonic scan that may be used to compute the locationrelationship between ultrasonic scans. So long as there is at least onesonogram image information frame that includes the two identifiedgraphical artifacts, the location relationship between the sonogramimage information from all of the ultrasonic scans can be computed.

FIG. 11 illustrates an alternative embodiment of the ultrasonagraphicsystem 100 that employs an electromagnetic (E/M) sensor 1102. The E/Msensor 1102 is placed on the head of the ultrasound transducer probe 104and optionally on the patient (not shown) to track to positions relativeto each other. Alternatively, the patient can be kept still and just usethe ultrasound tracking E/M sensor 102 relative to the room or the spacein which it is placed. Accordingly, the ultrasound frame 1104 iscaptured and recorded in 3D space 1106. An electromagnetic transducer1108, at a known reference location and orientation, emits anelectromagnetic field that is detectable by the electromagnetic sensor1102. The detected electromagnetic field is used to determine thelocation and orientation of the ultrasound transducer probe 104.

FIG. 12 illustrates an alternative embodiment that employs an internalmoveable ultrasonic transmitter 1202 within the ultrasound transducerprobe 104. The automatically moving ultrasonic transmitter 1202 sweepsin different directions to generate 3D image and/or 3D model data basedon the sonogram image information acquired as the ultrasonic transmitter1202 sweeps across the human subject 102 (FIG. 1 ). FIGS. 13 a-13 dillustrate various sweep patterns that a moveable ultrasonic transmitter1202 might make. In FIG. 13 a the moveable ultrasonic transmitter 1202moves in an arc pattern by some predefined angle between ultrasoundscans. In FIG. 13 b the moveable ultrasonic transmitter 1202 rotates 180degrees by some predefined angle between ultrasound scans. In FIG. 13 cthe moveable ultrasonic transmitter 1202 moves laterally by some lateraldistance between ultrasound scans. FIG. 13 d the moveable ultrasonictransmitter 1202 moves by some predefined angular displacement betweenultrasound scans.

With such embodiments, the AI algorithm can be used to improve accuracyin the generation of 3D image and/or 3D model data from a series ofsonogram image information 1302 acquired as the transmitter 1202automatically moves in proximity to the scanned tissue of interest 1304.Further, if the ultrasound transducer probe 104 is moved to a differentlocation on the human subject 102, MEMs devices may optionally be usedto determine location of the ultrasound transducer probe 104 for aseries of scans on the human subject 102.

In the field of cardiology and medical imaging, speckle trackingechocardio-graphy (STE) is an echocardiographic imaging technique. STEanalyzes the motion of tissues in the heart or other organs by using thenaturally occurring speckle pattern in the myocardium (or in the motionof blood) when imaged by ultrasound using the ultrasound transducerprobe 104. The STE method of documentation of myocardial motion is anoninvasive method of definition for both vectors and velocity. Here,the series of sonogram image information captured along one scan plane,followed by a series of sonogram image information captured along thenext scan plane can be used to construct time variant generate 3D imageand/or 3D model data. Here, the practitioner will be able to view imagesthat corresponds to a 3D movie.

In some embodiments, two or more of the above-described features may beused together to improve accuracy of the generated 3D image and/or 3Dmodel data.

It should be emphasized that the above-described embodiments of theultrasonagraphic system 100 are merely possible examples ofimplementations of the invention. Many variations and modifications maybe made to the above-described embodiments. All such modifications andvariations are intended to be included herein within the scope of thisdisclosure and protected by the following claims.

Furthermore, the disclosure above encompasses multiple distinctinventions with independent utility. While each of these inventions hasbeen disclosed in a particular form, the specific embodiments disclosedand illustrated above are not to be considered in a limiting sense asnumerous variations are possible. The subject matter of the inventionsincludes all novel and non-obvious combinations and subcombinations ofthe various elements, features, functions and/or properties disclosedabove and inherent to those skilled in the art pertaining to suchinventions. Where the disclosure or subsequently filed claims recite “a”element, “a first” element, or any such equivalent term, the disclosureor claims should be understood to incorporate one or more such elements,neither requiring nor excluding two or more such elements.

Applicant(s) reserves the right to submit claims directed tocombinations and subcombinations of the disclosed inventions that arebelieved to be novel and non-obvious. Inventions embodied in othercombinations and subcombinations of features, functions, elements and/orproperties may be claimed through amendment of those claims orpresentation of new claims in the present application or in a relatedapplication. Such amended or new claims, whether they are directed tothe same invention or a different invention and whether they aredifferent, broader, narrower or equal in scope to the original claims,are to be considered within the subject matter of the inventionsdescribed herein.

Therefore, having thus described the invention, at least the followingis claimed:
 1. An ultrasonagraphic three dimensional (3D) modelgeneration system, comprising: an ultrasound image data processorcommunicatively coupled to an ultrasound transducer probe that isconfigured to acquire sonogram image information from a serial pluralityof ultrasound scans of a human subject, wherein the ultrasoundtransducer probe generates the sonogram image information during eachone of the plurality of ultrasound scans, wherein the ultrasoundtransducer probe receives the sonogram image information from theultrasound transducer probe, and wherein the ultrasound image dataprocessor generates a plurality of time indexed sonogram imageinformation portions that each include the sonogram image informationand a first time that corresponds to a time of acquisition of thesonogram image information; at least one image capture device thatcaptures a time sequenced series of camera images that include both thehuman subject and the ultrasound transducer probe, wherein at least onefirst optical target that is located on the human subject is visible ineach one of the series of captured camera images, and wherein at leastone second optical target that is located on the ultrasound transducerprobe is visible in each one of the series captured camera images; anoptical tracker unit communicatively coupled to the at least one imagecapture device, wherein the optical tracker unit receives each one ofthe series of camera images from the at least one image capture device,wherein each one of the series of camera images includes a time thatindicates a time of acquisition of the camera image by the at least oneimage capture device; wherein the optical tracker unit determines alocation of the at least one first target on the human subject and alocation of the at least one second target on the ultrasound transducerprobe, wherein the optical tracker unit determines time indexed locationinformation for each camera image based on the determined locations ofthe first optical target and the second optical target, wherein the timeindexed location information identifies a portion of the human subjectthat is being scanned by the ultrasound transducer probe based on thedetermined location of the at least one optical target on the humansubject and the at least one second optical target on the ultrasoundtransducer probe, and wherein the time indexed location informationincludes the second time; an image registration module that iscommunicatively coupled to the ultrasound image data processor and theat least one image capture device, wherein the image registration moduleselects, for each time indexed sonogram image information portion, thetime indexed location information that has the same second time, andwherein the image registration module generates a plurality of locationindexed sonogram information portions based on the each one of the timeindexed sonogram image information portions and the corresponding timeindexed location information; and an image processing algorithm modulethat is communicatively coupled to the image registration module,wherein the image processing algorithm module receives the plurality oflocation indexed sonogram image information portions from the imageregistration module, and wherein the image processing algorithm modulecombines the received plurality of location indexed sonogram imageinformation portions to generate a composite sonogram image information,wherein a 3D model is generated based on the generated compositesonogram image information.
 2. The ultrasonagraphic 3D model generationsystem of claim 1, wherein the image processing algorithm modulegenerates 3D composite sonogram graphical information based on thegenerated composite sonogram image information.
 3. The ultrasonagraphic3D model generation system of claim 2, further comprising: a 3Dvisualization module that generates an image based on the 3D compositesonogram graphical information, wherein the image is presentable on adisplay.
 4. The ultrasonagraphic 3D model generation system of claim 1,further comprising: a 3D visualization module that generates an imagebased on the 3D model, wherein the image is presentable on a display. 5.The ultrasonagraphic 3D model generation system of claim 1, wherein theimage processing algorithm module generates 3D composite sonogramgraphical information based on the generated composite sonogram imageinformation model, and wherein the generated 3D composite sonogramgraphical information is defined as a collection of points in a 3D spacecorresponding to a surface of an organ or interest.
 6. Theultrasonagraphic 3D model generation system of claim 5, wherein thecollection of points in the 3D space is a point cloud of geometricsamples on the surface of the organ of interest.
 7. The ultrasonagraphic3D model generation system of claim 1, further comprising: at least oneclock, wherein the at least one clock provides the time of acquisitionof the ultrasound information by the ultrasound transducer probe,wherein the at least one clock provides the time of acquisition of thecamera image that is captured by the at least one image capture device.8. A method of generating three dimensional (3D) model data based on aserial plurality of ultrasound scans of a human subject that areacquired using an ultrasonagraphic system, comprising: generating aplurality of time indexed sonogram image information portions that eachinclude sonogram image information from a portion of the ultrasound scanand a first time that corresponds to a time of acquisition of thesonogram image information portion; capturing a time sequenced series ofcamera images that include both the human subject and the ultrasoundtransducer probe using at least one image capture device, wherein eachone of the series of camera images includes a second time that indicatesa time of acquisition of the camera image by the at least one imagecapture device; determining time indexed location information for eachcamera image, wherein the time indexed location information identifies alocation of a portion of the human subject that is being scanned by theultrasound transducer probe, and wherein the time indexed locationinformation includes the associated second time; selecting, for eachtime indexed sonogram image information portion, the time indexedlocation information that has the same second time as the first time ofthe sonogram image information portion; generating a plurality oflocation indexed sonogram information portions based on the each one ofthe time indexed sonogram image information portions and thecorresponding time indexed location information; combining the receivedplurality of location indexed sonogram image information portions togenerate a composite sonogram image information; and generating the 3Dmodel data based on the generated composite sonogram image information.9. The method of claim 8, wherein at least one first optical target islocated on the human subject and is visible in each one of the series ofcaptured camera images, wherein at least one second optical target thatis located on the ultrasound transducer probe is visible in each one ofthe series captured camera images, wherein determining time indexedlocation information for each camera image comprises: determining alocation of the at least one first target on the human subject and alocation of the at least one second target on the ultrasound transducerprobe, and wherein the location of the portion of the human subject thatis being scanned by the ultrasound transducer probe is determined basedon the determined location of the at least one first target and thedetermined location of the at least one second target.
 10. The method ofclaim 8, wherein generating a plurality of time indexed sonogram imageinformation portions comprises: adding the first time to the each one ofthe time indexed sonogram image information portions using a clock of anultrasound image data processor; and adding the second time to the eachone of the camera images using the clock of the ultrasound image dataprocessor that receives the serial plurality of camera images.
 11. Themethod of claim 8, further comprising: generating 3D composite sonogramgraphical information based on the generated composite sonogram imageinformation.
 12. The method of claim 11, generating a 3D image based onthe 3D composite sonogram graphical information; and presenting the 3Dimage on a display.
 13. The method of claim 8, generating a 3D imagebased on the 3D model data; and presenting the 3D image on a display.14. The method of claim 8, wherein the generated 3D model data isdefined as a collection of points in a 3D space corresponding to asurface of an organ or interest.
 15. The method of claim 14, wherein thecollection of points in the 3D space is a point cloud of geometricsamples on the surface of the organ of interest.
 17. The method of claim14, wherein the organ of interest is at least a portion of a spine ofthe human subject.
 18. The method of claim 17, wherein generating the 3Dmodel data comprises: generating a graphical artifact that indicates thedetermined at least one of position and orientation between twovertebrae of the spine; and presenting the generated graphical artifacton the presented image of the spine.
 19. The method of claim 14,filtering the sonogram image information in each received portion of theultrasound scan to discriminate the organ of interest from othersurrounding organs, wherein sonogram image information associated withthe other surrounding organs is removed.